Camera module and depth map extraction method thereof

ABSTRACT

A camera module according to one embodiment of the present invention comprises: an illumination unit for outputting a signal of incident light irradiated to an object; a lens unit for collecting a signal of reflection light reflected from the object, an image sensor unit for generating an electric signal from a reflection light signal collected by the lens unit, a tilting unit for shifting an optical path of the reflection light signal, and an image control unit for extracting a depth map of the object by using a phase difference between the incident light signal with respect to a frame having shifted by the tilting unit and the reflection light signal received by the image sensor unit, wherein the lens unit is disposed on the image sensor unit and includes an infrared (IR) filter disposed on the image sensor unit and at least one lens disposed on the infrared filter, and the tilting unit controls tilt of the infrared filter.

TECHNICAL FIELD

The present invention relates to a camera module and a depth mapextraction method thereof.

BACKGROUND ART

Three-dimensional (3D) content is applied not only to games and culturebut also to many fields such as education, manufacturing, and autonomousdriving, and a depth map is required to acquire 3D content. The depthmap is information indicating a distance in space and representsperspective information of one point with respect to another point of atwo-dimensional (2D) image.

One method of acquiring a depth map includes projecting an infrared (IR)structured light onto an object, interpreting light reflected from theobject, and extracting a depth map. The IR structured light scheme has aproblem in that it is difficult to obtain a desired level of depthresolution for a moving object.

Meanwhile, a time-of-flight (ToF) scheme has attracted attention as atechnology that replaces the IR structured light scheme.

According to the ToF scheme, the distance to an object is calculated bymeasuring a flight time, that is, a time for light to be shot,reflected, and return. The greatest advantage of the ToF scheme is thatit quickly provides distance information regarding 3D space in realtime. Also, a user can obtain accurate distance information withoutseparate algorithm application or hardware correction. Also, a user canacquire an accurate depth map even if he or she measures a very closesubject or a moving subject.

However, the current ToF scheme has a problem in that informationobtainable per frame, i.e., resolution, is low.

One way to increase resolution is to increase the number of pixels of animage sensor. However, in this case, there is a problem in that thevolume and production cost of a camera module increase significantly.

Accordingly, there is a need for a depth map acquisition method capableof increasing resolution without significantly increasing the volume andproduction cost of a camera module.

DISCLOSURE Technical Problem

An object of the present invention is to provide a camera module forextracting a depth map using a time-of-flight (ToF) scheme and a depthmap extraction method thereof.

Technical Solution

According to an embodiment of the present invention, there is provided acamera module including a lighting unit configured to output an incidentlight signal to be emitted to an object, a lens unit configured tocollect a reflected light signal reflected from the object, an imagesensor unit configured to generate an electric signal from the reflectedlight signal collected by the lens unit, a tilting unit configured toshift an optical path of the reflected light signal, and an imagecontrol unit configured to extract a depth map of the object from aframe shifted by the tilting unit using a phase difference between theincident light signal and the reflected light signal received by theimage sensor unit, wherein the lens unit is disposed on the image sensorunit, the lens unit comprises an infrared (IR) filter disposed on theimage sensor unit and at least one lens disposed on the IR filter, andthe tilting unit controls a slope of the IR filter.

The tilting unit may include a voice coil motor (VCM), and the IR filtermay be disposed between the image sensor unit and the VCM.

The VCM may include a magnet holder, a plurality of magnets disposed onthe magnet holder and spaced apart from one another at predeterminedintervals, a coil holder, and a plurality of coils disposed on the coilholder and spaced apart from one another at predetermined intervals tomake pairs with the plurality of magnets.

The IR filter may include a glass layer and a glass layer holderconfigured to support the glass layer, and at least a portion of theglass layer holder may be surrounded by the magnet holder.

The magnet holder may include a plurality of magnet guides foraccommodating the plurality of magnets, the glass layer holder mayinclude a plurality of protrusions corresponding to the plurality ofmagnet guides, and the plurality of protrusions may be moved to bebrought into contact with or spaced apart from the plurality of magnetguides according to a magnetic field generated between the plurality ofcoils and the plurality of magnets.

The glass layer may be tilted to a predetermined angle according tomovement of the plurality of protrusions.

The glass layer may be an IR pass glass layer.

The IR filter may further include an IR pass glass layer disposed on theimage sensor unit and spaced apart from the glass layer.

The camera module may further include an elastic film disposed betweenthe image sensor and the IR filter.

The camera module may further include a housing configured toaccommodate the image sensor, and the elastic film may be adhered to thehousing.

According to another embodiment of the present invention, there isprovided a camera module including a lighting unit configured to outputan incident light signal to be emitted to an object, a lens unitconfigured to collect a reflected light signal reflected from theobject, an image sensor unit configured to generate an electric signalfrom the reflected light signal collected by the lens unit, an elasticfilm disposed on the image sensor unit, a tilting unit configured toshift an optical path of the reflected light signal, and an imagecontrol unit configured to extract a depth map of the object from aframe shifted by the tilting unit using a phase difference between theincident light signal and the reflected light signal received by thesensor unit, wherein the tilting unit controls a shape of the elasticfilm.

The camera module may further include a housing configured toaccommodate the image sensor, and one face of the elastic film may becoupled to the housing, and the other face of the elastic film may becoupled to the tilting unit.

Advantageous Effects

With the camera module according to an embodiment of the presentinvention, it is possible to acquire a depth map with high resolutionwithout significantly increasing the number of pixels of an imagesensor.

Also, according to an embodiment, it is possible to obtain a subpixelshift effect using a simple structure, and also it is possible toprotect an image sensor from moisture, foreign matter, and the like.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a time-of-flight (ToF) camera moduleaccording to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a frequency of an incident lightsignal.

FIG. 3 is an example cross-sectional view of a camera module.

FIG. 4 is a diagram illustrating an electric signal generation processaccording to an embodiment of the present invention.

FIG. 5 is a diagram illustrating an image sensor 130 according to anembodiment of the present invention.

FIG. 6 is a diagram illustrating that a tilting unit 140 changes anoptical path of a reflected light signal.

FIGS. 7 and 8 are diagrams illustrating an SR technique according to anembodiment of the present invention.

FIG. 9 is a diagram illustrating a pixel value arrangement processaccording to an embodiment of the present invention.

FIGS. 10 and 11 are diagrams illustrating an effect of shifting an imageframe input to an image sensor by controlling a slope of an infrared(IR) filter.

FIG. 12 is a perspective view of a voice coil motor (VCM) and an IRfilter according to an embodiment of the present invention.

FIG. 13 is a cross-sectional view of a ToF camera module including a VCMand an IR filter according to an embodiment of the present invention.

FIG. 14 is a diagram showing a process of coupling an IR filter and amagnet assembly included in a VCM according to an embodiment of thepresent invention.

FIG. 15 is a diagram showing a coupling process of a coil assemblyincluded in a VCM according to an embodiment of the present invention.

FIG. 16 is a diagram showing a process of coupling a magnet assembly, anIR filter, and a coil assembly according to an embodiment of the presentinvention.

FIG. 17 is a cross-sectional view of a portion of a camera moduleaccording to an embodiment of the present invention.

FIGS. 18 to 23 show various examples of placing an elastic film.

MODE FOR CARRYING OUT THE INVENTION

While the invention is susceptible to various modifications and may haveseveral embodiments, specific embodiments thereof are shown by way ofexample in the drawings and will be described herein. It should beunderstood, however, that there is no intent to limit the invention tothe particular forms disclosed, but on the contrary, the invention is tocover all modifications, equivalents, and alternatives falling withinthe spirit and scope of the invention.

Although the terms “first,” “second,” etc. may be used herein todescribe various elements, these elements should not be limited by theseterms. These terms are only used to distinguish one element fromanother. For example, a first element may be called a second element,and a second element may also be called a first element withoutdeparting from the scope of the present invention. As used herein, theterm “and/or” includes any and all combinations of one or more of theassociated listed items.

It will be understood that when an element is referred to as being“connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a,” “an,” and “one” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises,”“comprising,” “includes,” and/or “including,” when used herein, specifythe presence of stated features, integers, steps, operations, elements,components, and/or combinations thereof but do not preclude the presenceor addition of one or more other features, integers, steps, operations,elements, components, and/or combinations thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art andwill not be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Hereinafter, exemplary embodiments will be described in detail withreference to the accompanying drawings. The same reference numerals aregiven to the same or equivalent elements throughout the drawings andredundant descriptions thereof will be omitted.

FIG. 1 is a block diagram of a ToF camera module according to anembodiment of the present invention.

Referring to FIG. 1, the ToF camera module 100 includes a lighting unit110, a lens unit 120, an image sensor unit 130, a tilting unit 140, andan image control unit 150.

The lighting unit 110 generates an incident light signal and emits thegenerated incident light signal to an object. In this case, the lightingunit 110 may generate and output an incident light signal in the form ofa pulse wave or a continuous wave. The continuous wave may be asinusoidal wave or a squared wave. By generating an incident lightsignal in the form of a pulse wave or a continuous wave, the ToF cameramodule 100 may detect a phase difference between an incident lightsignal output from the lighting unit 110 and a reflected light signalreflected from an object. Herein, incident light may refer to light thatis output from the lighting unit 110 and incident on an object, andreflected light may refer to light that is output from the lighting unit110 and then reflected from an object after reaching the object. Fromthe position of the ToF camera module 100, the incident light may beoutput light, and the reflected light may be incident light.

The lighting unit 110 may emit the generated incident light signal tothe object during a predetermined integration time. Here, theintegration time refers to one frame period. When a plurality of framesare generated, the predetermined integration time is repeated. Forexample, when the ToF camera module 100 captures the object at 20 fps,the integration time is 1/20 sec. Also, when 100 frames are generated,the integration time may be repeated 100 times.

The lighting unit 110 may generate a plurality of incident light signalshaving different frequencies. The lighting unit 110 may sequentially andrepeatedly generate a plurality of incident light signals havingdifferent frequencies. Alternatively, the lighting unit 110 may generatea plurality of incident light signals having different frequencies atthe same time.

FIG. 2 is a diagram illustrating a frequency of an incident lightsignal. According to an embodiment of the present invention, thelighting unit 110 may control the first half of the integration time togenerate an incident light signal having a frequency f₁ and may controlthe other half of the integration time to generate an incident lightsignal having a frequency f₂.

According to another embodiment, the lighting unit 110 may control someof a plurality of light-emitting diodes to generate an incident lightsignal having a frequency f₁ and may control the other light-emittingdiodes to generate an incident light signal having a frequency f₂.

To this end, the lighting unit 110 may include a light source 112configured to generate light and a light modulating unit 114 configuredto modulate light.

First, the light source 112 generates light. The light generated by thelight source 112 may be infrared light having a wavelength of 770 nm to3000 nm or may be visible light having a wavelength of 380 to 770 nm.The light source 112 may use a light-emitting diode (LED) and may have aplurality of LEDs arranged in a certain pattern. In addition, the lightsource 112 may include an organic light-emitting diode (OLED) or a laserdiode (LD).

The light source 112 is repeatedly turned on and off at predeterminedtime intervals to generate an incident light signal in the form of apulse wave or a continuous wave. A predetermined time interval may bethe frequency of the incident light signal. The turning-on and off ofthe light source may be controlled by the light modulating unit 114.

The light modulating unit 114 controls the turning-on and off of thelight source 112 to control the light source 112 to generate an incidentlight signal in the form of a continuous wave or a pulse wave. The lightmodulating unit 114 may control the light source 112 to generate anincident light signal in the form of a continuous wave or a pulse wavethrough frequency modulation or pulse modulation.

Meanwhile, the lens unit 120 collects a reflected light signal reflectedfrom the object and forwards the reflected light signal to the imagesensor unit 130.

FIG. 3 is an example cross-sectional view of a camera module.

Referring to FIG. 3, the camera module 300 includes a lens assembly 310,an image sensor 320, and a printed circuit board 330. Here, the lensassembly 310 may correspond to the lens unit 120 of FIG. 1, and theimage sensor 320 may correspond to the image sensor unit 130 of FIG. 1.Also, the image control unit 150 and the like of FIG. 1 may beimplemented in the printed circuit board 330. Although not shown, thelighting unit 110 of FIG. 1 may be disposed on the side of the imagesensor 320 on the printed circuit board 330 or may be disposed outsidethe camera module 300.

The lens assembly 310 may include a lens 312, a lens barrel 314, a lensholder 316, and an IR filter 318.

The lens 312 may include a plurality of lens and may include one lens.When the lens 312 includes a plurality of lens, the lens may be arrangedwith respect to a central axis to form an optical system. Here, thecentral axis may be the same as an optical axis of the optical system.

The lens barrel 314 is coupled to the lens holder 316 to provide a spacefor accommodating lens. The lens barrel 314 may be rotatably coupled toone or a plurality of lenses, but this is just an example. Therefore,the lens barrel 314 and the lenses may be coupled in another way, suchas a scheme using an adhesive (e.g., an adhesive resin such as epoxy).

The lens holder 316 may be coupled to the lens barrel 314 to support thelens barrel 314 and may be coupled to the printed circuit board 330equipped with the image sensor 320. The lens holder 316 may form a spacefor attachment of the IR filter 318 under the lens barrel 314. A helicalpattern may be formed on an inner circumferential surface of the lensholder 316, and similarly, a helical pattern may be formed on an outercircumferential surface of the lens barrel 314. Thus, the lens holder316 and the lens barrel 314 may be rotatably coupled to each other.However, this is just an example, and the lens holder 316 and the lensbarrel 314 may be coupled to each other through an adhesive or may beintegrally formed.

The lens holder 316 may include an upper holder 316-1 to be coupled tothe lens barrel 314 and a lower holder 316-2 to be coupled to theprinted circuit board 330 equipped with the image sensor 320. The upperholder 316-1 and the lower holder 316-2 may be formed integrally witheach other, may be separated but can be engaged with or coupled to eachother, or may be separated and spaced apart from each other. In thiscase, the upper holder 316-1 may have a smaller diameter than the lowerholder 316-2.

The above example is just an embodiment, and the lens unit 120 may beconfigured in another structure capable of collecting a reflected lightsignal incident on the ToF camera module 100 and forwarding thereflected light signal to the image sensor unit 130.

Referring to FIG. 1 again, the image sensor unit 130 generates anelectric signal using the reflected light signal collected through thelens unit 120.

The image sensor unit 130 may be synchronized with the turning-on andoff period of the lighting unit 110 to absorb the reflected lightsignal. In detail, the image sensor unit 130 may absorb the light inphase or out of phase with the incident light signal output from thelighting unit 110. That is, the image sensor unit 130 may repeatedlyperform a step of absorbing a reflected light signal while the lightsource is turned on and a step of absorbing a reflected light signalwhile the light source is turned off.

Subsequently, the image sensor unit 130 may use a plurality of referencesignals with different phase differences to generate an electric signalcorresponding to each reference signal. The frequency of the referencesignal may be set to be the same as the frequency of the incident lightsignal output from the lighting unit 110. Accordingly, when the lightingunit 110 generates incident light signals using a plurality offrequencies, the image sensor unit 130 generates electric signals usinga plurality of reference signals corresponding to the frequencies. Theelectric signals may include information regarding electric chargequantities or voltages corresponding to the reference signals.

FIG. 4 is a diagram illustrating an electric signal generation processaccording to an embodiment of the present invention.

As shown in FIG. 4, the reference signal according to an embodiment ofthe present invention may include four reference signals C₁ to C₄. Thereference signals C₁ to C₄ may have the same frequency as the incidentlight signal and have a phase difference of 90 degrees from one another.The reference signal C₁, which is one of the four reference signals, mayhave the same phase as the incident light signal. A reflected lightsignal has a phase delayed by a distance traveled by an incident lightsignal incident on and returned from an object. The image sensor unit130 mixes the reflected light signal with each of the reference signals.Thus, the image sensor unit 130 may generate an electric signalcorresponding to a shaded portion of FIG. 4 for each reference signal.

In another embodiment, when incident light signals are generated using aplurality of frequencies during an integration time, the image sensorunit 130 absorbs reflected light signals corresponding to the pluralityof frequencies. For example, it is assumed that incident light signalshaving frequencies f₁ and f₂ are generated, and the plurality ofreference signals have a phase difference of 90 degrees from oneanother. In this case, reflected light signals also have frequencies f₁and f₂. Thus, four electric signals may be generated using the reflectedlight signal with the frequency f₁ and corresponding four referencesignals. Also, four electric signals may be generated using thereflected light signal with the frequency f₂ and corresponding fourreference signals. Accordingly, a total of eight electric signals may begenerated.

The image sensor unit 130 may be configured in a structure in which aplurality of pixels are arranged in a grid form. The image sensor unit130 may be a complementary metal oxide semiconductor (CMOS) image sensoror a charged coupled device (CCD) image sensor. Also, the image sensorunit 130 may include a ToF sensor configured to receive infrared lightreflected from a subject and measure a distance from the subject using atraveled time or a phase difference.

FIG. 5 is a diagram illustrating an image sensor 130 according to anembodiment of the present invention. For example, for an image sensor130 with a resolution of 320×240 as shown in FIG. 5, 76,800 pixels arearranged in a grid form. In this case, a predetermined interval, such asa shaded portion of FIG. 5, may be formed between the plurality ofpixels. In an embodiment of the present invention, one pixel refers to apixel and a predetermined interval adjacent to the pixel.

According to an embodiment of the present invention, each pixel 132 mayinclude a first light receiving unit 132-1 including a first photodiodeand a first transistor and a second light receiving unit 132-2 includinga second photodiode and a second transistor.

The first light receiving unit 132-1 receives a reflected light signalin the same phase as the waveform of the incident light. That is, whilethe light source is turned on, the first photodiode is turned on toabsorb a reflected light signal. Also, while the light source is turnedoff, the first photodiode is turned off to stop absorbing a reflectedlight signal. The first photodiode converts the absorbed reflected lightsignal into an electric current and forwards the electric current to thefirst transistor. The first transistor converts the forwarded electriccurrent into an electric signal and outputs the electric signal.

The second light receiving unit 132-2 receives a reflected light signalin the opposite phase to the waveform of the incident light. That is,while the light source is turned on, the second photodiode is turned offto absorb a reflected light signal. Also, while the light source isturned off, the second photodiode is turned on to stop absorbing areflected light signal. The second photodiode converts the absorbedreflected light signal into an electric current and forwards theelectric current to the second transistor. The second transistorconverts the forwarded electric current into an electric signal.

Thus, the first light receiving unit 132-1 may be referred to as anin-phase receiving unit, and the second light receiving unit 132-2 maybe referred to as an out-of-phase receiving unit. When, as describedabove, the first receiving unit 132-1 and the second light receivingunit 132-2 are activated with a time difference, the amount of lightreceived may vary depending on the distance to the object. For example,when the object is in front of the ToF camera module 100 (i.e., thedistance is equal to zero), the time it takes for light to be reflectedfrom the object after the light is output from the lighting unit 110 iszero, and thus the turning-on and off period of the light source becomesa light receiving period with no changes. Accordingly, only the firstlight receiving unit 132-1 can receive light, and the second lightreceiving unit 132-2 cannot receive light. As another example, when theobject is located a predetermined distance away from the ToF cameramodule 100, it takes time for light to be reflected from the objectafter the light is output from the lighting unit 110, and thus theturning-on and off period of the light source becomes different from alight receiving period. Accordingly, the amount of light received by thefirst light receiving unit 132-1 becomes different from that of thesecond light receiving unit 132-2. That is, the distance to the objectmay be calculated using the difference between the amount of light inputto the first light receiving unit 132-1 and the amount of light input tothe second light receiving unit 132-2. Referring to FIG. 1 again, theimage control unit 150 calculates a phase difference between incidentlight and reflected light using an electric signal received from theimage sensor unit 130 and calculates a distance between the object andthe ToF camera module 100 using the phase difference.

In detail, the image control unit 150 may calculate a phase differencebetween incident light and reflected light using information regardingelectric charge quantity of the electric signal.

As described above, four electric signals may be generated for eachfrequency of the incident light signal. Therefore, the image controlunit 150 may compute a phase difference t_(d) between the incident lightsignal and the reflected light signal using Equation 1 below:

$\begin{matrix}{t_{d} = {\arctan \left( \frac{Q_{3} - Q_{4}}{Q_{1} - Q_{2}} \right)}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

where Q₁ to Q₄ are electric charge quantities of four electric signals.Q₁ is an electric charge quantity of an electric signal corresponding toa reference signal having the same phase as the incident light signal.Q₂ is an electric charge quantity of an electric signal corresponding toa reference signal having a phase lagging by 180 degrees from theincident light signal. Q₃ is an electric charge quantity of an electricsignal corresponding to a reference signal having a phase lagging by 90degrees from the incident light signal. Q₄ is an electric chargequantity of an electric signal corresponding to a reference signalhaving a phase lagging by 270 degrees from the incident light signal.

Thus, the image control unit 150 may calculate a distance between theobject and the ToF camera module 100 using the phase difference betweenthe incident light signal and the reflected light signal. In this case,the image control unit 150 may compute a distance d between the objectand the ToF camera module 100 using Equation 2 below:

$\begin{matrix}{d = {\frac{c}{2f}\frac{t_{d}}{2\pi}}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

where c is the speed of light, and f is the frequency of incident light.

Meanwhile, in an embodiment of the present invention, a super resolution(SR) technique is used to increase the resolution of a depth map. The SRtechnique is a technique for obtaining a high-resolution image from aplurality of low-resolution images, and a mathematical model of the SRtechnique may be expressed using Equation 3 below:

y _(k) =D _(k) B _(k) M _(k) x+n _(k)   Equation 3

where 1≤k≤p, p is the number of low-resolution images, y_(k) is alow-resolution image (=y_(k,1), y_(k,2), . . . , y_(k,M) ^(T); here,M=N₁*N₂), D_(k) is a down sampling matrix, B_(k) is an optical blurmatrix, M_(k) is an image warping matrix, x is a high-resolution image(=x₁, x₂, . . . , x_(N) ^(T); here, N=L₁N₁*L₂N₂), and n_(k) is noise.That is, the SR technique refers to a technique for estimating x byapplying the inverse function of the estimated resolution degradationfactors to y_(k). The SR technique may be largely divided into astatistical scheme and a multi-frame scheme, and the multi-frame schememay be largely divided into a space division scheme and a time divisionscheme. When the SR technique is used to acquire a depth map, theinverse function of M_(k) of Equation 1 is not present, and thus thestatistical scheme may be tried. However, the statistical schemerequires a repeated computation process and thus has low efficiency.

In order to apply the SR technique to depth map extraction, the imagecontrol unit 150 may generate a plurality of low-resolution subframesusing an electric signal received from the image sensor unit 130 andthen may extract a plurality of low-resolution depth maps using theplurality of low-resolution subframes. Also, the image control unit 150may rearrange pixel values of the plurality of low-resolution depth mapsto extract a high-resolution depth map.

Here, the term “high resolution” has a relative meaning that representsa higher resolution than “low resolution.”

Here, the term “subframe” may refer to image data generated from anyintegration time and an electric signal corresponding to a referencesignal. For example, when an electric signal is generated using eightreference signals during a first integration time, i.e., one imageframe, eight subframes may be generated, and one start frame may befurther generated. Herein, a subframe may be used interchangeably withimage data, subframe image data, etc.

Alternatively, in order to apply the SR technique according to anembodiment of the present invention to depth map extraction, the imagecontrol unit 150 may generate a plurality of low-resolution subframesusing an electric signal received from the image sensor unit 130 andthen may rearrange pixel values of the plurality of low-resolutionsubframes to generate a plurality of high-resolution subframes. Also,the image control unit 150 may extract a high-resolution depth map usingthe high-resolution subframes.

To this end, a pixel shift technique may be used. That is, the imagecontrol unit 150 may acquire several sheets of image data shifted by asubpixel for each subframe using the pixel shift technique, acquire aplurality of pieces of high-resolution subframe image data by applyingthe SR technique for each subframe, and extract a high-resolution depthmap using the high-resolution subframe image data. In order to performpixel shift, the ToF camera module 100 according to an embodiment of thepresent invention includes the tilting unit 140.

Referring to FIG. 1 again, the tilting unit 140 changes an optical pathof at least one of an incident light signal or a reflected light signalin units of subpixels of the image sensor unit 130.

For each image frame, the tilting unit 140 changes an optical path of atleast one of an incident light signal or a reflected light signal. Asdescribed above, one image frame may be generated at every integrationtime. Accordingly, when one integration time ends, the tilting unit 140changes an optical path of at least one of an incident light signal or areflected light signal.

The tilting unit 140 changes an optical path of an incident light signalor a reflected light signal in units of subpixels with respect to theimage sensor unit 130. In this case, the tilting unit 140 changes anoptical path of at least one of an incident light signal or a reflectedlight signal upward, downward, leftward or rightward with respect to thecurrent optical path.

FIG. 6 is a diagram illustrating that the tilting unit 140 changes anoptical path of a reflected light signal.

In FIG. 6A, a portion indicated by solid lines indicates a currentoptical path of the reflected light signal, and a portion indicated bydotted lines indicates a changed optical path. When an integration timecorresponding to the current optical path ends, the tilting unit 140 maychange the optical path of the reflected light signal as represented bydotted lines. Thus, the path of the reflected light signal is shifted bya subpixel from the current optical path. For example, as shown in FIG.6A, when the tilting unit 140 shifts the current optical path to theright by 0.173 degrees, the reflected light signal incident on the imagesensor unit 130 may be shifted to the right by 0.5 pixels (subpixels).

According to an embodiment of the present invention, the tilting unit140 may change an optical path of a reflected light signal clockwisewith respect to a reference position. For example, as shown in FIG. 6B,after a first integration time ends, the tilting unit 140 shifts theoptical path of the reflected light signal to the right by 0.5 pixelswith respect to the image sensor unit 130 during a second integrationtime. Also, the tilting unit 140 shifts the optical path of thereflected light signal downward by 0.5 pixels with respect to the imagesensor unit 130 during a third integration time. Also, the tilting unit140 shifts the optical path of the reflected light signal leftward by0.5 pixels with respect to the image sensor unit 130 during a fourthintegration time. Also, the tilting unit 140 shifts the optical path ofthe reflected light signal upward by 0.5 pixels with respect to theimage sensor unit 130 during a fifth integration time. That is, thetilting unit 140 may shift the optical path of the reflected lightsignal to its original position during four integration times. This canbe applied in the same way even when an optical path of an incidentlight signal is shifted, and a detailed description thereof will beomitted. Also, the optical path change pattern being clockwise is justan example, and the optical path change pattern may be counterclockwise.

Meanwhile, the subpixel may be greater than zero pixels and smaller thanone pixel. For example, the subpixel may have a size of 0.5 pixels andmay have a size of ⅓ pixels. The size of the subpixel can be changed indesign by a person skilled in the art.

FIGS. 7 and 8 are diagrams illustrating an SR technique according to anembodiment of the present invention.

Referring to FIG. 7, the image control unit 150 may extract a pluralityof low-resolution depth maps using a plurality of low-resolutionsub-frames generated during the same integration time, i.e., during thesame frame. Also, the image control unit 150 may rearrange pixel valuesof the plurality of low-resolution depth maps to extract ahigh-resolution depth map. Here, optical paths of incident light signalsor reflected light signals corresponding to the plurality oflow-resolution depth maps may be different from each other.

For example, the image control unit 150 may generate low-resolutionsubframes 1-1 to 4-8 using a plurality of electric signals.Low-resolution subframes 1-1 to 1-8 are low-resolution subframesgenerated during the first integration time. Low-resolution subframes2-1 to 2-8 are low-resolution subframes generated during the secondintegration time. Low-resolution subframes 3-1 to 3-8 are low-resolutionsubframes generated during the third integration time. Low-resolutionsubframes 4-1 to 4-8 are low-resolution subframes generated during thefourth integration time. Thus, the image control unit 150 applies adepth map extraction technique to the plurality of low-resolutionsubframes generated for each integration time to extract low-resolutiondepth maps LRD-1 to LRD-4. Low-resolution depth map LRD-1 is alow-resolution depth map extracted using subframes 1-1 to 1-8.Low-resolution depth map LRD-2 is a low-resolution depth map extractedusing subframes 2-1 to 2-8. Low-resolution depth map LRD-3 is alow-resolution depth map extracted using subframes 3-1 to 3-8.Low-resolution depth map LRD-4 is a low-resolution depth map extractedusing subframes 4-1 to 4-8. Also, the image control unit 150 rearrangespixel values of low-resolution depth maps LRD-1 to LRD-4 to extracthigh-resolution depth map HRD.

Alternatively, as described above, the image control unit 150 mayrearrange pixel values of a plurality of subframes corresponding to thesame reference signal to generate a high-resolution subframe. In thiscase, the plurality of subframes have different optical paths ofcorresponding incident light signals or reflected light signals. Also,the image control unit 150 may extract a high-resolution depth map usinga plurality of high-resolution subframes.

For example, as shown in FIG. 8, the image control unit 150 generateslow-resolution subframes 1-1 to 4-8 using a plurality of electricsignals. Low-resolution subframes 1-1 to 1-8 are low-resolutionsubframes generated during the first integration time. Low-resolutionsubframes 2-1 to 2-8 are low-resolution subframes generated during thesecond integration time. Low-resolution subframes 3-1 to 3-8 arelow-resolution subframes generated during the third integration time.Low-resolution subframes 4-1 to 4-8 are low-resolution subframesgenerated during the fourth integration time. Here, low-resolutionsubframes 1-1, 2-1, 3-1, and 4-1 correspond to the same reference signalC₁ and different optical paths. Then, the image control unit 150 mayrearrange pixel values of low-resolution subframes 1-1, 2-1, 3-1, and4-1 to generate high-resolution subframe H-1. When high-resolutionsubframes H1 to H8 are generated through the rearrangement of the pixelvalues, the image control unit may apply the depth map extractiontechnique to high-resolution subframes H-1 to H-8 to extract ahigh-resolution depth map HRD.

FIG. 9 is a diagram illustrating a pixel value arrangement processaccording to an embodiment of the present invention.

Here, it is assumed that four low-resolution images having a size of 4×4are used to generate one high-resolution image having a size of 8x 8. Inthis case, the high-resolution pixel grid has 8×8 pixels, which are thesame as pixels of a high-resolution image. Here, the low-resolutionimage may have a meaning including a low-resolution subframe and alow-resolution depth map, and the high-resolution image may have ameaning including a high-resolution subframe and a high-resolution depthmap.

In FIG. 9, first to fourth low-resolution images are images capturedwhen an optical path is shifted in units of a subpixel with a 0.5-pixelsize. The image control unit 150 arranges pixel values of the second tofourth low-resolution images to fit the high-resolution image in adirection in which the optical path is shifted with respect to the firstlow-resolution image in which the optical path is not shifted.

In detail, the second low-resolution image is an image shifted to theright by a subpixel from the first low-resolution image. Therefore, apixel B of the second low-resolution image is arranged in a pixellocated to the right of each pixel A of the first low-resolution image.

The third low-resolution image is an image shifted downward by asubpixel from the second low-resolution image. Therefore, a pixel C ofthe third low-resolution image is arranged in a pixel located under eachpixel B of the second low-resolution image.

The fourth low-resolution image is an image shifted to the left by asubpixel from the third low-resolution image. Therefore, a pixel D ofthe fourth low-resolution image is arranged in a pixel located to theleft of the pixel C of the third low-resolution image.

When all pixel values of the first to fourth low-resolution images arerearranged in a high-resolution pixel grid, a high-resolution imageframe which has a resolution four times that of a low-resolution imageis generated.

Meanwhile, the image control unit 150 may apply a weight value to anarranged pixel value. In this case, the weight value may be setdifferently depending on the size of the subpixel or the shift directionof the optical path and may be set differently for each low-resolutionimage.

To this end, the tilting unit 140 may change the optical path throughsoftware or hardware. The amount of calculation of the ToF camera module100 increases when the tilting unit 140 changes the optical path throughsoftware, and the ToF camera module 100 becomes complicated in structureor increases in volume when the tilting unit 140 change the optical paththrough hardware.

According to an embodiment of the present invention, the tilting unit140 obtains data shifted by a subpixel using a method of controlling theslope of a lens assembly, e.g., an IR filter 318 (see FIG. 2) includedin the lens assembly.

FIGS. 10 and 11 are diagrams illustrating an effect of shifting an imageframe input to an image sensor by controlling the slope of an IR filter.FIG. 11 shows a result of simulating a distance shifted for a tiltingangle under the condition that the thickness of the IR filter is 0.21 mmand that the refractive index of IR is 1.5.

Referring to FIG. 10 and the following Equation 4, the shifted distanceand the slope θ₁ of the IR filter 318 may have the followingrelationship.

$\begin{matrix}{{\Delta \; x} = {d\; \cos \; {\theta_{1}\left( {\frac{1}{\tan \left( {{90{^\circ}} - \theta_{1}} \right)} - \frac{1}{\tan \left( {{90{^\circ}} - \theta_{2}} \right)}} \right)}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

where θ₂ may be expressed using Equation 5 below:

$\begin{matrix}{\theta_{2} = {\sin^{- 1}\left( \frac{\sin \; \theta_{1}}{n_{g}} \right)}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

where θ₁ is the slope of the IR filter 318, i.e., a tilting angle of theIR filter 318, n_(g) is the refractive index of the IR filter 318, and dis the thickness of the IR filter 318. For example, referring toEquations 4 and 5, the IR filter 318 may be tilted by about 5 to 6degrees in order to shift an image frame input to the image sensor by 7μm. In this case, the vertical displacement of the IR filter 318 may beabout 175 to 210 μm.

When the slope of the IR filter 318 is controlled as described above, itis possible to obtain shifted image data without tilting the imagesensor 320.

According to an embodiment of the present invention, the tilting unitfor controlling the slope of the IR filter may include a voice coilmotor (VCM), and the IR filter 318 may be disposed between the imagesensor and the VCM.

FIG. 12 is a perspective view of a VCM and an IR filter according to anembodiment of the present invention, FIG. 13 is a cross-sectional viewof a ToF camera module including a VCM and an IR filter according to anembodiment of the present invention, FIG. 14 is a diagram showing aprocess of coupling an IR filter and a magnet assembly included in a VCMaccording to an embodiment of the present invention, FIG. 15 is adiagram showing a coupling process of a coil assembly included in a VCMaccording to an embodiment of the present invention, and FIG. 16 is adiagram showing a process of coupling a magnet assembly, an IR filter,and a coil assembly according to an embodiment of the present invention.

Referring to FIGS. 12 to 16, the tilting unit 140 may include a VCM1000, and the VCM 1000 may include a magnet assembly 1100 and a coilassembly 1200 and may be coupled to, brought into contact with, orconnected to the IR filter 318.

In FIG. 13A, for convenience of description, it is shown that the VCM1000 is surrounded by the lens barrel 314 and the lens holder 318 andthat the lens 312 and the IR filter 318 are omitted. However, the lens312 and the IR filter 318 may be arranged as shown in FIG. 3. That is,the lens 312 may be surrounded by the lens barrel 314 or may beaccommodated in a space of the VCM 1000. Alternatively, the lens barrel314 may be an element of the VCM 1000.

According to an embodiment of the present invention, the magnet assembly1100 may include a magnet holder 1110 and a plurality of magnets 1120,and the plurality of magnets 1120 may be spaced apart on the magnetholder 1110 at predetermined intervals. For example, the magnet holder1110 may have a hollow circular ring shape or a quadrilateral ringshape, and a plurality of magnet guides 1112 may be formed toaccommodate the plurality of magnets 1120.

Here, the magnet holder 1110 may contain a magnetic material or a softmagnetic material, e.g., Fe.

Subsequently, the coil assembly 1200 may include a coil holder 1210, aplurality of coils 1220, and a coil terminal 1230, and the plurality ofcoils 1220 may be disposed on the coil holder 1210 and spaced apart fromone another at predetermined intervals to make pairs with the pluralityof magnets. For example, the coil holder 1210 may have a hollow circularring shape or a quadrilateral ring shape, and a plurality of coil guides1212 may be formed to accommodate the plurality of coils 1220. The coilholder 1210 may be the lens barrel 314. The coil terminal 1230 may beconnected to the plurality of coils 1220 and may apply power to theplurality of coils 1220.

The IR filter 318 includes a glass layer holder 3182 and a glass layer3184 supported by the glass layer holder 3182. The glass layer holder3182 may include a first glass layer holder 3182-1 disposed under theglass layer 3184 and a second glass layer holder 3182-2 disposed on anupper edge of the glass layer 3184. The second glass layer holder 3182-2may have a hollow circular ring shape or a quadrilateral ring shape andmay be disposed in a hollow of the magnet holder 1110 and surrounded bythe magnet holder 1110. In this case, the second glass layer holder3182-2 may include a plurality of protrusions P1, P2, P3, and P4corresponding to the plurality of magnet guides 1112 of the magnetholder 1110. The plurality of protrusions P1, P2, P3, and P4 may bemoved such that the protrusions are brought into contact with or spacedapart from the plurality of magnet guides 1112. The second glass layerholder 3182-2 may contain a magnetic material or a soft magneticmaterial.

When power is applied to the plurality of coils 1220 through the coilterminal 1230, an electric current flows through the plurality of coils1220, and thus it is possible to generate a magnetic field between theplurality of coils 1220 and the plurality of magnets 1120.

Thus, an electric driving force may be generated between the pluralityof magnet guides 1112 and the plurality of protrusions P1, P2, P3, andP4 of the second glass layer holder 3182-2, and the glass layer 3184supported by the second glass layer holder 3182-2 may be tilted at apredetermined angle.

For example, a slope formed between the protrusion P1 and the protrusionP3 or a slope formed between the protrusion P2 and the protrusion P4 mayvary depending on a force applied between the plurality of magnet guides1112 and the plurality of protrusions P1, P2, P3, and P4.

Also, the slope of the glass layer 3184 may vary depending on the slopeformed between the protrusion P1 and the protrusion P3 or the slopeformed between the protrusion P2 and the protrusion P4. Here, the slopeof the IR filter 318, and particularly, the slope of the glass layer3184 varies depending on the positions of the plurality of protrusionsP1, P2, P3, and P4 of the second glass layer holder 3182-2. Accordingly,the second glass layer holder 3182-2 may be referred to herein as ashaper.

In this case, for the degree of freedom of tilting of the glass layer3184, a spacer 1130 may be further disposed between the magnet holder1110 and the first glass layer holder 3182-1.

Here, the glass layer 3184 may be an IR-pass glass layer.

Alternatively, as shown in FIG. 13B, the glass layer 3184 may be ageneral glass layer, and the IR filter 318 may further include an IRpass glass layer 3186 spaced apart from the glass layer 3184 anddisposed on the image sensor 320. When the IR pass glass layer 3186 isdisposed on the image sensor 320, it is possible to reduce thepossibility of moisture or foreign matter directly penetrating into theimage sensor 320.

Meanwhile, according to an embodiment of the present invention, themagnet assembly 1110 may further include a magnet holder 1140. Themagnet holder 1140 may support upper portions of the plurality ofmagnets 1120, and thus the plurality of magnets 1120 may move morestably and reliably.

As described above, according to an embodiment of the present invention,the slope of the IR filter 318 may be controlled according to thedriving of the VCM 1000. To this end, the IR filter 318 should bedisposed together with the VCM 1000, and thus the IR filter 318 needs tobe spaced apart from the image sensor 320.

Meanwhile, according to an embodiment of the present invention, theslope of the IR filter 318 needs to be frequently changed, and thus afree space for the movement of the IR filter 318 is required. In thiscase, the possibility of moisture, foreign matter, and the likepenetrating into the free space for the movement of the IR filter 318increases, and thus the image sensor 320 may be easily exposed tomoisture or foreign matter.

In an embodiment of the present invention, a component for preventingthe image sensor 320 from being exposed to moisture, foreign matter, andthe like may be further included.

FIG. 17 is a cross-sectional view of a portion of a camera moduleaccording to an embodiment of the present invention. Here, forconvenience of description, an upper portion of the camera module, e.g.,the lens, the lens barrel, the VCM, and the like are omitted, but thedescription of FIGS. 3 and 10 to 14 may be equally applied.

Referring to FIG. 17, an image sensor 320 may be mounted on a printedcircuit board 330 and accommodated in a housing 340. Here, the housingmay be a second lens holder 316-2. The slope of the IR filter 318 may becontrolled by the VCM 1000 (see FIGS. 12 to 16). For example, when afirst protrusion P1 of a first glass layer holder 3182-1 faces upwardand the third protrusion P3 faces downward due to the driving of the VCM1000, a glass layer 3184 of an IR filter 318 may be tilted.

According to an embodiment of the present invention, an elastic film1400 may be disposed between the IR filter 318 and the image sensor 320.The elastic film 1400 may be fastened to the housing 340. In this case,one face of the elastic film 1400 may be fastened to the housing 340,and the other face of the elastic film 1400 may be coupled to thetilting unit 140. The elastic film 1400 may be, for example, a reverseosmosis (RO) membrane, a nano filtration (NF) membrane, anultra-filtration (UF) membrane, a micro filtration (MF) membrane, or thelike. Here, the RO membrane is a membrane having a pore size of about 1to 15 angstroms, the NF membrane is a membrane having a pore size ofabout 10 angstroms, the UF membrane is a membrane having a pore size ofabout 15 to 200 angstroms, and the MF membrane is a membrane having apore size of about 200 to 1000 angstroms. Accordingly, it is possible toprevent moisture, foreign matter, and the like from penetrating into aspace between the IR filter 318 and the housing 340, that is, the spacearranged for the movement of the IR filter 318.

In this case, the elastic film 1400 may be a transparent and stretchablefilm with a thickness of 25 to 50 and the IR filter 318 may be disposedon the elastic film 1400 so that at least a portion of the IR filter 318can be in direct contact with the elastic film 1400. That is, the shapeof the elastic film 1400 may be controlled by the tilting unit 1400.Thus, when the IR filter 318 is inclined, the elastic film 1400 may bestretched or contracted together with the IR filter 318. When the IRfilter 318 returns to its original position, the elastic film 1400 maybe restored immediately along with the IR filter 318. Accordingly, it ispossible to stably support the movement of the IR filter 318.

FIGS. 18 to 23 show various examples of placing an elastic film.

Referring to FIG. 18, the elastic film 1400 may be adhered to thehousing 340 for accommodating the image sensor 320 through an adhesive1410.

Referring to FIG. 19, the elastic film 1400 may be fastened to thehousing 340 for housing the image sensor 320 through an instrument 1420.

Referring to FIG. 20, the elastic film 1400 may be disposed to cover theouter circumferential surface of the housing 340 for accommodating theimage sensor 320. In order to fasten the elastic film 1400, anadditional fastening member 1430 may be disposed to surround the outercircumferential surface of the housing 340.

Referring to FIG. 21, the elastic film 1400 may be disposed directly onthe image sensor 320.

Referring to FIG. 22, the elastic film 1400 may be disposed between thefirst glass layer holder 3182-1 and the housing 340 and fastened byinstructions 1440 and 1442.

Referring to FIG. 23, the elastic film 1400 may be adhered to the firstglass layer holder 3182-1 and the housing 340 through adhesives 1450 and1452.

While the present invention has been described with reference toexemplary embodiments, these are just examples and do not limit thepresent invention. It will be understood by those skilled in the artthat various modifications and applications may be made therein withoutdeparting from the essential characteristics of the embodiments. Forexample, elements described in the embodiments above in detail may bemodified and implemented. Furthermore, differences associated with suchmodifications and applications should be construed as being included inthe scope of the present invention defined by the appended claims.

DESCRIPTION OF THE SYMBOLS 100: ToF camera module 110: lighting unit120: lens unit 130: image sensor unit 140: tilting unit 150: imagecontrol unit

1. A camera module comprising: a lighting unit configured to output anincident light signal to be emitted to an object; a lens unit configuredto collect a reflected light signal reflected from the object; an imagesensor unit configured to generate an electric signal from the reflectedlight signal collected by the lens unit; a tilting unit configured toshift an optical path of the reflected light signal; and an imagecontrol unit configured to extract a depth information of the objectfrom a frame shifted by the tilting unit using a phase differencebetween the incident light signal and the reflected light signalreceived by the image sensor unit, wherein the lens unit is disposed onthe image sensor unit, the lens unit comprises an infrared (IR) filterdisposed on the image sensor unit and at least one lens disposed on theIR filter, and the tilting unit controls a slope of the IR filter. 2.The camera module of claim 1, wherein the tilting unit comprises a voicecoil motor (VCM), and the IR filter is disposed between the image sensorunit and the VCM.
 3. The camera module of claim 2, wherein the VCMcomprises: a magnet holder; a plurality of magnets disposed on themagnet holder and spaced apart from one another at predeterminedintervals; a coil holder; and a plurality of coils disposed on the coilholder and spaced apart from one another at predetermined intervals tomake pairs with the plurality of magnets.
 4. The camera module of claim3, wherein the IR filter comprises a glass layer and a glass layerholder configured to support the glass layer, and at least a portion ofthe glass layer holder is surrounded by the magnet holder.
 5. The cameramodule of claim 4, wherein the magnet holder comprises a plurality ofmagnet guides for accommodating the plurality of magnets, the glasslayer holder comprises a plurality of protrusions corresponding to theplurality of magnet guides, and the plurality of protrusions are movedto be brought into contact with or spaced apart from the plurality ofmagnet guides according to a magnetic field generated between theplurality of coils and the plurality of magnets.
 6. The camera module ofclaim 5, wherein the glass layer is tilted to a predetermined angleaccording to movement of the plurality of protrusions.
 7. The cameramodule of claim 4, wherein the glass layer is an IR pass glass layer. 8.The camera module of claim 4, wherein the IR filter comprises an IR passglass layer disposed on the image sensor unit and spaced apart from theglass layer.
 9. A camera module comprising: a lighting unit configuredto output an incident light signal to be emitted to an object; a lensunit configured to collect a reflected light signal reflected from theobject; an image sensor unit configured to generate an electric signalfrom the reflected light signal collected by the lens unit; an elasticfilm disposed on the image sensor unit; a tilting unit configured toshift an optical path of the reflected light signal; and an imagecontrol unit configured to extract a depth information of the objectfrom a frame shifted by the tilting unit using a phase differencebetween the incident light signal and the reflected light signalreceived by the sensor unit, wherein the tilting unit controls a shapeof the elastic film.
 10. The camera module of claim 9, comprising ahousing configured to accommodate the image sensor, wherein one face ofthe elastic film is coupled to the housing, and the other face of theelastic film is coupled to the tilting unit.
 11. The camera module ofclaim 1, wherein the tilting unit is configured to shift the opticalpath of the reflected light signal at every integration time.
 12. Thecamera module of claim 11, wherein the tilting unit is configured toshift the optical path of the reflected light signal in a unit ofsubpixel which is greater than 0 pixel and smaller than 1 pixel.
 13. Thecamera module of claim 11, wherein the tilting unit is configured toshift the optical path of the reflected light signal during a firstintegration time in a first direction, shift the optical path of thereflected light signal during a second integration time in a seconddirection, shift the optical path of the reflected light signal during athird integration time in a third direction, and shift the optical pathof the reflected light signal during a fourth integration time in afourth direction.
 14. The camera module of claim 11, wherein thelighting unit is configured to output a plurality of incident lightsignals having different frequencies.
 15. The camera module of claim 14,wherein the image sensor unit is configured to mix the reflected lightsignal with a plurality of reference signals having a predeterminedphase differences.
 16. The camera module of claim 15, wherein the imagecontrol unit is configured to extract the depth information using aplurality of subframes generated in different integration times.
 17. Thecamera module of claim 15, wherein the image control unit is configuredto extract the depth information using a plurality of subframesgenerated in different reference signals.
 18. The camera module of claim1, comprising an elastic film disposed between the image sensor and theIR filter.
 19. The camera module of claim 18, comprising a housingconfigured to accommodate the image sensor, wherein the elastic film isadhered to the housing.
 20. The camera module of claim 9, wherein thetilting unit is configured to shift the optical path of the reflectedlight signal at every integration time.